feat(orchestrator): re-plan the queue from the Speculator each run - #506
Merged
Conversation
behinddwalls
force-pushed
the
preetam/speculation-run
branch
from
August 4, 2026 04:46
8bd6b25 to
f9ddb16
Compare
behinddwalls
marked this pull request as ready for review
August 4, 2026 04:47
This was referenced Aug 4, 2026
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 4, 2026 05:11
f2e6633 to
dda3dd3
Compare
behinddwalls
force-pushed
the
preetam/speculation-run
branch
from
August 4, 2026 05:11
f9ddb16 to
55134c7
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 5, 2026 03:15
dda3dd3 to
96feaf5
Compare
behinddwalls
force-pushed
the
preetam/speculation-run
branch
from
August 5, 2026 03:15
55134c7 to
344901c
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 5, 2026 03:26
96feaf5 to
d450509
Compare
behinddwalls
force-pushed
the
preetam/speculation-run
branch
from
August 5, 2026 03:26
344901c to
40fe38a
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 5, 2026 03:38
d450509 to
9227c40
Compare
behinddwalls
force-pushed
the
preetam/speculation-run
branch
from
August 5, 2026 03:38
40fe38a to
2bd693a
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 5, 2026 03:44
9227c40 to
dbd61ed
Compare
behinddwalls
force-pushed
the
preetam/speculation-run
branch
from
August 5, 2026 03:44
2bd693a to
a7b44df
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 5, 2026 04:11
dbd61ed to
baf15bf
Compare
behinddwalls
force-pushed
the
preetam/speculation-run
branch
from
August 5, 2026 04:12
a7b44df to
329e026
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 5, 2026 04:57
baf15bf to
82985ad
Compare
behinddwalls
force-pushed
the
preetam/speculation-run
branch
from
August 5, 2026 04:57
329e026 to
2d5e6e8
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 6, 2026 02:15
64f7e91 to
d51250d
Compare
behinddwalls
force-pushed
the
preetam/speculation-run
branch
from
August 6, 2026 02:15
110416b to
fd31568
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 6, 2026 02:34
d51250d to
650a0d2
Compare
behinddwalls
force-pushed
the
preetam/speculation-run
branch
2 times, most recently
from
August 6, 2026 03:04
99a2c76 to
dcf9b20
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
2 times, most recently
from
August 6, 2026 03:17
a351b6b to
04e65c5
Compare
behinddwalls
force-pushed
the
preetam/speculation-run
branch
from
August 6, 2026 03:17
dcf9b20 to
1087b03
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 6, 2026 03:32
04e65c5 to
c167e22
Compare
behinddwalls
force-pushed
the
preetam/speculation-run
branch
from
August 6, 2026 03:32
1087b03 to
b9f222a
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 7, 2026 03:15
c167e22 to
b2e22cc
Compare
behinddwalls
force-pushed
the
preetam/speculation-run
branch
from
August 7, 2026 03:15
b9f222a to
8c9e75e
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 7, 2026 03:30
b2e22cc to
10ac3fc
Compare
behinddwalls
force-pushed
the
preetam/speculation-run
branch
from
August 7, 2026 03:30
8c9e75e to
d549bd7
Compare
mnoah1
approved these changes
Aug 7, 2026
mnoah1
left a comment
Contributor
There was a problem hiding this comment.
Left a few things but nothing major if you want to take a look later (other than the publishBatchID one).
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 7, 2026 19:48
10ac3fc to
fc33bc5
Compare
behinddwalls
force-pushed
the
preetam/speculation-run
branch
from
August 7, 2026 19:48
d549bd7 to
a6ab892
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 7, 2026 19:54
fc33bc5 to
b8131db
Compare
behinddwalls
force-pushed
the
preetam/speculation-run
branch
from
August 7, 2026 19:54
a6ab892 to
0cb75fb
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 7, 2026 20:13
b8131db to
17bbf48
Compare
behinddwalls
force-pushed
the
preetam/speculation-run
branch
from
August 7, 2026 20:13
0cb75fb to
830ba89
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 7, 2026 20:14
17bbf48 to
288ee84
Compare
behinddwalls
force-pushed
the
preetam/speculation-run
branch
from
August 7, 2026 20:15
830ba89 to
8d5a6c2
Compare
behinddwalls
changed the base branch from
preetam/speculation-cancel-msgid
to
main
August 7, 2026 20:23
behinddwalls
force-pushed
the
preetam/speculation-run
branch
from
August 7, 2026 20:23
8d5a6c2 to
d346213
Compare
## Summary ### Why? The speculate controller had no speculation in it: it advanced one batch at a time along a single hard-coded chain — every dependency assumed to pass — and nothing ever called the Speculator, the Generator, or the Allocator that had been built for it. This wires that machinery into the pipeline: a queue's paths are chosen by a swappable extension within a build budget, and each is built against only the dependencies it assumes will succeed. ### What? Every message is a dirty signal naming a batch; the controller re-plans that batch's whole queue from a single read: read the state, cancel paths whose assumptions a finished dependency has proven wrong, ask the Speculator, filter its proposals, dispatch what survives. Nothing carries over between runs, so duplicated or reordered signals are harmless and a later run repairs whatever an earlier one left half-done. `doc.go` explains the model in plain terms — no vocabulary section, every term is defined where it is used. The path set keeps exactly one writer — this run. The build stages record what CI did on per-build records; the run folds those into the set and alone decides each path's status. Pending paths are re-dispatched every run until their build is seen running, dispatches partition by batch so heads proceed in parallel, and cancelling paths need no dispatch at all: the poll loop reads the stop off the set and enacts it. Speculation lands inert. The wiring layer passes a placeholder Speculator that proposes nothing, so the run executes end to end but funds no paths; composing real per-queue speculators and turning the feature on is the wiring change at the top of this stack. Batch outcomes still come from the legacy per-batch finalizer, which waits on every dependency — strictly stricter than path-aware finalization, so the system stays correct until the next commit replaces it. ## Test Plan ✅ `bazel test //submitqueue/orchestrator/controller/speculate/...` — assumption checks and proposal filtering are table-driven; run tests cover funding a first path, re-dispatching pending paths, broken-path cancellation, build results recorded onto paths, lost CAS races skipped rather than failed, and Speculator errors abandoning the run. ✅ `make fmt`, `make gazelle` # Conflicts: # service/submitqueue/orchestrator/server/BUILD.bazel # service/submitqueue/orchestrator/server/main.go # submitqueue/orchestrator/controller/speculate/BUILD.bazel # submitqueue/orchestrator/controller/speculate/speculate.go # submitqueue/orchestrator/controller/speculate/speculate_test.go # Please enter the commit message for your changes. Lines starting # with '#' will be kept; you may remove them yourself if you want to. # An empty message aborts the commit. # # interactive rebase in progress; onto 452922a9 # Last command done (1 command done): # pick b9f222a # feat(orchestrator): re-plan the queue from the Speculator each run # No commands remaining. # You are currently rebasing branch 'preetam/speculation-run' on '452922a9'. # # Changes to be committed: # modified: service/submitqueue/orchestrator/server/BUILD.bazel # modified: service/submitqueue/orchestrator/server/main.go # modified: submitqueue/orchestrator/BUILD.bazel # modified: submitqueue/orchestrator/controller/speculate/BUILD.bazel # new file: submitqueue/orchestrator/controller/speculate/check.go # new file: submitqueue/orchestrator/controller/speculate/check_test.go # new file: submitqueue/orchestrator/controller/speculate/dispatch.go # new file: submitqueue/orchestrator/controller/speculate/doc.go # new file: submitqueue/orchestrator/controller/speculate/run.go # new file: submitqueue/orchestrator/controller/speculate/run_test.go # new file: submitqueue/orchestrator/controller/speculate/snapshot.go # new file: submitqueue/orchestrator/controller/speculate/snapshot_test.go # modified: submitqueue/orchestrator/controller/speculate/speculate.go # modified: submitqueue/orchestrator/controller/speculate/speculate_test.go # modified: submitqueue/orchestrator/pipeline.go # # Conflicts: # service/submitqueue/orchestrator/server/main.go # Please enter the commit message for your changes. Lines starting # with '#' will be kept; you may remove them yourself if you want to. # An empty message aborts the commit. # # interactive rebase in progress; onto fc33bc5 # Last command done (1 command done): # pick d549bd7 # feat(orchestrator): re-plan the queue from the Speculator each run # No commands remaining. # You are currently rebasing branch 'preetam/speculation-run' on 'fc33bc54'. # # Changes to be committed: # modified: service/submitqueue/orchestrator/server/BUILD.bazel # modified: service/submitqueue/orchestrator/server/main.go # modified: submitqueue/orchestrator/BUILD.bazel # modified: submitqueue/orchestrator/controller/speculate/BUILD.bazel # new file: submitqueue/orchestrator/controller/speculate/check.go # new file: submitqueue/orchestrator/controller/speculate/check_test.go # new file: submitqueue/orchestrator/controller/speculate/dispatch.go # new file: submitqueue/orchestrator/controller/speculate/doc.go # new file: submitqueue/orchestrator/controller/speculate/run.go # new file: submitqueue/orchestrator/controller/speculate/run_test.go # new file: submitqueue/orchestrator/controller/speculate/snapshot.go # new file: submitqueue/orchestrator/controller/speculate/snapshot_test.go # modified: submitqueue/orchestrator/controller/speculate/speculate.go # modified: submitqueue/orchestrator/controller/speculate/speculate_test.go # modified: submitqueue/orchestrator/pipeline.go #
behinddwalls
force-pushed
the
preetam/speculation-run
branch
from
August 7, 2026 20:37
d346213 to
643005d
Compare
behinddwalls
temporarily deployed
to
stack-rebase
August 7, 2026 20:51 — with
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why?
The speculate controller had no speculation in it: it advanced one batch at a time along a single hard-coded chain — every dependency assumed to pass — and nothing ever called the Speculator, the Generator, or the Allocator that had been built for it. This wires that machinery into the pipeline: a queue's paths are chosen by a swappable extension within a build budget, and each is built against only the dependencies it assumes will succeed.
What?
Every message is a dirty signal naming a batch; the controller re-plans that batch's whole queue from a single read: read the state, cancel paths whose assumptions a finished dependency has proven wrong, ask the Speculator, filter its proposals, dispatch what survives. Nothing carries over between runs, so duplicated or reordered signals are harmless and a later run repairs whatever an earlier one left half-done.
doc.goexplains the model in plain terms — no vocabulary section, every term is defined where it is used.The path set keeps exactly one writer — this run. The build stages record what CI did on per-build records; the run folds those into the set and alone decides each path's status. Pending paths are re-dispatched every run until their build is seen running, dispatches partition by batch so heads proceed in parallel, and cancelling paths need no dispatch at all: the poll loop reads the stop off the set and enacts it.
Speculation lands inert. The wiring layer passes a placeholder Speculator that proposes nothing, so the run executes end to end but funds no paths; composing real per-queue speculators and turning the feature on is the wiring change at the top of this stack. Batch outcomes still come from the legacy per-batch finalizer, which waits on every dependency — strictly stricter than path-aware finalization, so the system stays correct until the next commit replaces it.
Test Plan
✅
bazel test //submitqueue/orchestrator/controller/speculate/...— assumption checks and proposal filtering are table-driven; run tests cover funding a first path, re-dispatching pending paths, broken-path cancellation, build results recorded onto paths, lost CAS races skipped rather than failed, and Speculator errors abandoning the run.✅
make fmt,make gazelleIssues